PCS: Predictive Component-level Scheduling for Reducing Tail Latency in Cloud Online Services

摘要

Modern latency-critical online services often rely on composing results froma large number of server components. Hence the tail latency (e.g. the 99thpercentile of response time), rather than the average, of these componentsdetermines the overall service performance. When hosted on a cloud environment,the components of a service typically co-locate with short batch jobs toincrease machine utilizations, and share and contend resources such as cachesand I/O bandwidths with them. The highly dynamic nature of batch jobs in termsof their workload types and input sizes causes continuously changingperformance interference to individual components, hence leading to theirlatency variability and high tail latency. However, existing techniques eitherignore such fine-grained component latency variability when managing serviceperformance, or rely on executing redundant requests to reduce the taillatency, which adversely deteriorate the service performance when load getsheavier. In this paper, we propose PCS, a predictive and component-levelscheduling framework to reduce tail latency for large-scale, parallel onlineservices. It uses an analytical performance model to simultaneously predict thecomponent latency and the overall service performance on different nodes. Basedon the predicted performance, the scheduler identifies straggling componentsand conducts near-optimal component-node allocations to adapt to the changingperformance interferences from batch jobs. We demonstrate that, using realisticworkloads, the proposed scheduler reduces the component tail latency by anaverage of 67.05\% and the average overall service latency by 64.16\% comparedwith the state-of-the-art techniques on reducing tail latency.